In [ ]:
import pandas as pd

In [ ]:
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
try:
    import seaborn
except ImportError:
    pass

Tabular data


In [ ]:
df = pd.read_csv("data/titanic.csv")

In [ ]:
df.head()

Starting from reading this dataset, to answering questions about this data in a few lines of code:

What is the age distribution of the passengers?


In [ ]:
df['Age'].hist()

How does the survival rate of the passengers differ between sexes?


In [ ]:
df.groupby('Sex')[['Survived']].aggregate(lambda x: x.sum() / len(x))

Or how does it differ between the different classes?


In [ ]:
df.groupby('Pclass')['Survived'].aggregate(lambda x: x.sum() / len(x)).plot(kind='bar')

Are young people more likely to survive?


In [ ]:
df['Survived'].sum() / df['Survived'].count()

In [ ]:
df25 = df[df['Age'] <= 25]
df25['Survived'].sum() / len(df25['Survived'])

All the needed functionality for the above examples will be explained throughout this tutorial.

Data structures

Pandas provides two fundamental data objects, for 1D (Series) and 2D data (DataFrame).

Series

A Series is a basic holder for one-dimensional labeled data. It can be created much as a NumPy array is created:


In [ ]:
s = pd.Series([0.1, 0.2, 0.3, 0.4])
s

Attributes of a Series: index and values

The series has a built-in concept of an index, which by default is the numbers 0 through N - 1


In [ ]:
s.index

You can access the underlying numpy array representation with the .values attribute:


In [ ]:
s.values

We can access series values via the index, just like for NumPy arrays:


In [ ]:
s[0]

Unlike the NumPy array, though, this index can be something other than integers:


In [ ]:
s2 = pd.Series(np.arange(4), index=['a', 'b', 'c', 'd'])
s2

In [ ]:
s2['c']

In this way, a Series object can be thought of as similar to an ordered dictionary mapping one typed value to another typed value.

In fact, it's possible to construct a series directly from a Python dictionary:


In [ ]:
pop_dict = {'Germany': 81.3, 
            'Belgium': 11.3, 
            'France': 64.3, 
            'United Kingdom': 64.9, 
            'Netherlands': 16.9}
population = pd.Series(pop_dict)
population

We can index the populations like a dict as expected:


In [ ]:
population['France']

but with the power of numpy arrays:


In [ ]:
population * 1000

DataFrames: Multi-dimensional Data

A DataFrame is a tablular data structure (multi-dimensional object to hold labeled data) comprised of rows and columns, akin to a spreadsheet, database table, or R's data.frame object. You can think of it as multiple Series object which share the same index.

One of the most common ways of creating a dataframe is from a dictionary of arrays or lists.

Note that in the IPython notebook, the dataframe will display in a rich HTML view:


In [ ]:
data = {'country': ['Belgium', 'France', 'Germany', 'Netherlands', 'United Kingdom'],
        'population': [11.3, 64.3, 81.3, 16.9, 64.9],
        'area': [30510, 671308, 357050, 41526, 244820],
        'capital': ['Brussels', 'Paris', 'Berlin', 'Amsterdam', 'London']}
countries = pd.DataFrame(data)
countries

Attributes of the DataFrame

A DataFrame has besides a index attribute, also a columns attribute:


In [ ]:
countries.index

In [ ]:
countries.columns

To check the data types of the different columns:


In [ ]:
countries.dtypes

An overview of that information can be given with the info() method:


In [ ]:
countries.info()

Also a DataFrame has a values attribute, but attention: when you have heterogeneous data, all values will be upcasted:


In [ ]:
countries.values

If we don't like what the index looks like, we can reset it and set one of our columns:


In [ ]:
countries = countries.set_index('country')
countries

To access a Series representing a column in the data, use typical indexing syntax:


In [ ]:
countries['area']

Basic operations on Series/Dataframes

As you play around with DataFrames, you'll notice that many operations which work on NumPy arrays will also work on dataframes.


In [ ]:
# redefining the example objects

population = pd.Series({'Germany': 81.3, 'Belgium': 11.3, 'France': 64.3, 
                        'United Kingdom': 64.9, 'Netherlands': 16.9})

countries = pd.DataFrame({'country': ['Belgium', 'France', 'Germany', 'Netherlands', 'United Kingdom'],
        'population': [11.3, 64.3, 81.3, 16.9, 64.9],
        'area': [30510, 671308, 357050, 41526, 244820],
        'capital': ['Brussels', 'Paris', 'Berlin', 'Amsterdam', 'London']})

Elementwise-operations (like numpy)

Just like with numpy arrays, many operations are element-wise:


In [ ]:
population / 100

In [ ]:
countries['population'] / countries['area']

Alignment! (unlike numpy)

Only, pay attention to alignment: operations between series will align on the index:


In [ ]:
s1 = population[['Belgium', 'France']]
s2 = population[['France', 'Germany']]

In [ ]:
s1

In [ ]:
s2

In [ ]:
s1 + s2

Reductions (like numpy)

The average population number:


In [ ]:
population.mean()

The minimum area:


In [ ]:
countries['area'].min()

For dataframes, often only the numeric columns are included in the result:


In [ ]:
countries.median()
EXERCISE: Calculate the population numbers relative to Belgium

In [ ]:

EXERCISE: Calculate the population density for each country and add this as a new column to the dataframe.

In [ ]:


In [ ]:

Some other useful methods

Sorting the rows of the DataFrame according to the values in a column:


In [ ]:
countries.sort_values('density', ascending=False)

One useful method to use is the describe method, which computes summary statistics for each column:


In [ ]:
countries.describe()

The plot method can be used to quickly visualize the data in different ways:


In [ ]:
countries.plot()

However, for this dataset, it does not say that much:


In [ ]:
countries['population'].plot(kind='bar')

You can play with the kind keyword: 'line', 'bar', 'hist', 'density', 'area', 'pie', 'scatter', 'hexbin'

Importing and exporting data

A wide range of input/output formats are natively supported by pandas:

  • CSV, text
  • SQL database
  • Excel
  • HDF5
  • json
  • html
  • pickle
  • ...

In [ ]:
pd.read

In [ ]:
states.to

Other features

  • Working with missing data (.dropna(), pd.isnull())
  • Merging and joining (concat, join)
  • Grouping: groupby functionality
  • Reshaping (stack, pivot)
  • Time series manipulation (resampling, timezones, ..)
  • Easy plotting

There are many, many more interesting operations that can be done on Series and DataFrame objects, but rather than continue using this toy data, we'll instead move to a real-world example, and illustrate some of the advanced concepts along the way.

See the next notebooks!

Acknowledgement

© 2015, Stijn Van Hoey and Joris Van den Bossche (mailto:stijnvanhoey@gmail.com, mailto:jorisvandenbossche@gmail.com). Licensed under CC BY 4.0 Creative Commons

This notebook is partly based on material of Jake Vanderplas (https://github.com/jakevdp/OsloWorkshop2014).